Device and method for calibrating a camera of a vehicle

ABSTRACT

A device for calibrating a camera of a vehicle includes: a first camera for acquiring a first image, a second camera for acquiring a second image, and a processor that extracts a first class of interest from the first image, and extracts a second class of interest from the second image. The processor projects pixel coordinates of the first class of interest onto the second image to convert the pixel coordinates, and corrects parameters of the second camera such that a difference between the converted pixel coordinates of the first class of interest and pixel coordinates of the second class of interest is minimized.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of and priority to Korean PatentApplication No. 10-2022-0098725, filed in the Korean IntellectualProperty Office on Aug. 8, 2022, the entire contents of which areincorporated herein by reference.

FIELD

The present disclosure relates to a device and a method for calibratinga camera of a vehicle.

BACKGROUND

The statements in this section merely provide background informationrelated to the present disclosure and may not constitute prior art.

An autonomous vehicle uses an image acquired via a camera disposed onthe vehicle during travel control fused with a result of recognition bydeep learning. For this purpose, images acquired by a plurality ofcameras disposed on the vehicle must be calibrated to be converted intothe same real-world coordinates.

However, the cameras are located in a front portion and left and rightportions of the vehicle, so that the images acquired during travel aredifferent from each other and various obstacles exist in the acquiredimages. Thus, it is hard to find points desired to estimate arelationship between the plurality of cameras. Therefore, a technologydevelopment is desired to provide information necessary for the vehicletravel by correcting the plurality of images acquired by the pluralityof cameras on the autonomous vehicle to be converted into the same realword coordinates.

SUMMARY

The present disclosure has been made to solve the above-mentionedproblems occurring in the prior art while advantages achieved by theprior art are maintained intact.

An aspect of the present disclosure provides a device and a method forcalibrating a camera that correct a plurality of images acquired by aplurality of cameras to be converted into the same real-worldcoordinates.

The technical problems to be solved by the present disclosure are notlimited to the aforementioned problems, and any other technical problemsnot mentioned herein should be clearly understood from the followingdescription by those having ordinary skill in the art to which thepresent disclosure pertains.

According to an aspect of the present disclosure, a device forcalibrating a camera of a vehicle includes: a first camera for acquiringa first image, a second camera for acquiring a second image, and aprocessor that extracts a first class of interest from the first image,and extracts a second class of interest from the second image. Inparticular, the processor projects pixel coordinates of the first classof interest onto the second image to convert the pixel coordinates, andcorrects parameters of the second camera such that a difference betweenthe converted pixel coordinates of the first class of interest and pixelcoordinates of the second class of interest is minimized.

In one implementation, the processor may set a region including anobject having a height of zero “0” in a predetermined region of thefirst image as a region of interest, and extract pixels of the region ofinterest as the first class of interest.

In one implementation, the processor may set a region including anobject having a height of zero “0” in a predetermined region of thesecond image as a region of interest, and extract pixels of the regionof interest as the second class of interest.

In one implementation, the processor may extract a line or a patternpainted on a road surface as the first class of interest.

In one implementation, the processor may extract a line or a patternpainted on a road surface as the second class of interest.

In one implementation, the processor may convert the pixel coordinatesof the first class of interest by projecting the pixel coordinates ontothe second image using a warping function.

In one implementation, the first camera and the second camera may bedisposed at different positions.

According to another aspect of the present disclosure, a method forcalibrating a camera of a vehicle includes: acquiring a first image froma first camera; acquiring a second image from a second camera;extracting a first class of interest from the first image; andextracting a second class of interest from the second image. The methodfurther includes: projecting pixel coordinates of the first class ofinterest onto the second image to convert the pixel coordinates; andcorrecting parameters of the second camera such that a differencebetween the converted pixel coordinates of the first class of interestand pixel coordinates of the second class of interest is minimized.

In one implementation, the method may further include: setting a regionincluding an object having a height of zero “0” in a predeterminedregion of the first image as a region of interest, and extracting pixelsof the region of interest as the first class of interest.

In one implementation, the method may further include: setting a regionincluding an object having a height of zero “0” in a predeterminedregion of the second image as a region of interest, and extractingpixels of the region of interest as the second class of interest.

In one implementation, the method may further include extracting a lineor a pattern painted on a road surface as the first class of interest.

In one implementation, the method may further include extracting a lineor a pattern painted on a road surface as the second class of interest.

In one implementation, the method may further include converting thepixel coordinates of the first class of interest by projecting the pixelcoordinates onto the second image using a warping function.

In one implementation, the first camera and the second camera may bedisposed at different positions.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentdisclosure should be more apparent from the following detaileddescription taken in conjunction with the accompanying drawings:

FIG. 1 is a view showing a configuration of a camera calibration deviceof a vehicle according to an embodiment of the present disclosure;

FIGS. 2 to 5 are views showing a class of interest extracted accordingto an embodiment of the present disclosure;

FIG. 6 is a calibration graph of a second camera according to anembodiment of the present disclosure;

FIG. 7 is a view showing a change in matching of a first image and asecond image according to an embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating a camera calibration method of avehicle according to an embodiment of the present disclosure; and

FIG. 9 is a view showing a configuration of a computing system executinga method according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, some embodiments of the present disclosure are described indetail with reference to the exemplary drawings. In adding the referencenumerals to the components of each drawing, it should be noted that theidentical or equivalent component is designated by the identical numeraleven when they are displayed on other drawings. Further, in describingthe embodiment of the present disclosure, a detailed description of therelated known configuration or function has been omitted when it isdetermined that it interferes with the understanding of the embodimentof the present disclosure.

In describing the components of the embodiment according to the presentdisclosure, terms such as first, second, A, B, (a), (b), and the likemay be used. These terms are merely intended to distinguish thecomponents from other components, and the terms do not limit the nature,order or sequence of the components. Unless otherwise defined, all termsincluding technical and scientific terms used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich this disclosure belongs. It should be further understood thatterms, such as those defined in commonly used dictionaries, should beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art and should not be interpreted in anidealized or overly formal sense unless expressly so defined herein.

When a component, device, element, or the like of the present disclosureis described as having a purpose or performing an operation, function,or the like, the component, device, or element should be consideredherein as being “configured to” meet that purpose or to perform thatoperation or function.

FIG. 1 is a view showing a configuration of a camera calibration deviceof a vehicle according to an embodiment of the present disclosure.

As shown in FIG. 1 , a camera calibration device 100 of a vehicle mayinclude: a first camera 110, a second camera 120, a storage 130, and aprocessor 140.

According to an embodiment, the first camera 110 may be disposed in afront portion of the vehicle to acquire a first image including a frontimage of the vehicle. However, the present disclosure may not be limitedthereto. The first camera 110 may be disposed in a rear portion of thevehicle to acquire the first image including a rear image of thevehicle.

The second camera 120 may be disposed at a position different from thatof the first camera 110. According to an embodiment of the presentdisclosure, the second camera 120 may be disposed in a side portion ofthe vehicle to acquire a second image including a side image of thevehicle.

The storage 130 may store at least one algorithm for performingcalculation or execution of various commands for an operation of thecamera calibration device of the vehicle according to an embodiment ofthe present disclosure. The storage 130 may include at least one storagemedium among a flash memory, a hard disc, a memory card, a read-onlymemory (ROM), a random access memory (RAM), an electrically erasableprogrammable read-only memory (EEPROM), a programmable read-only memory(PROM), a magnetic memory, a magnetic disk, and an optical disk.

The processor 140 may extract a class of interest from the imagesacquired from the first camera 110 and the second camera 120. A moredetailed description is made with reference to FIGS. 2 to 5 .

FIGS. 2 to 5 are views showing a class of interest extracted accordingto an embodiment of the present disclosure.

As shown in FIG. 2 , the processor 140 may extract a first class ofinterest 20 from the first image acquired from the first camera 110.According to an embodiment, the processor 140 may set a region includingan object having a height of zero “0” in a predetermined region of thefirst image as a region of interest, and extract pixels of the region ofinterest as the first class of interest. As an example, the processor140 may extract a line or a pattern painted on a road surface as thefirst class of interest. The processor 140 may obtain pixel coordinatesof the first class of interest.

In addition, as shown in FIG. 3 , the processor 140 may extract a secondclass of interest 30 from the second image acquired from the secondcamera 120. According to an embodiment, the processor 140 may set aregion including an object having a height of zero “0” in apredetermined region of the second image as the region of interest, andextract pixels of the region of interest as the second class ofinterest. As an example, the processor 140 may extract the line or thepattern painted on the road surface as the second class of interest. Theprocessor 140 may obtain pixel coordinates of the second class ofinterest.

As shown in FIG. 4 , the processor 140 may assign a background class 22to pixels of a region excluding pixels of a region of interest in apredetermined region “A” of the first image acquired from the firstcamera 110.

In addition, as shown in FIG. 5 , the processor 140 may assign abackground class 32 to pixels of a region excluding pixels of a regionof interest in a predetermined region “B” of the second image acquiredfrom the second camera 120.

The processor 140 may convert the pixel coordinates of the first classof interest extracted from the first image by projecting the pixelcoordinates onto the second image. According to an embodiment, theprocessor 140 may convert the pixel coordinates of the first class ofinterest by projecting the pixel coordinates onto the second image usinga warping function “W”.

W(u _(front) , v _(front) , p)=h _(side)(h ⁻¹ _(front)(u _(front) , v_(front)))   <Calculation Formula 1>

where, (u_(front), V_(front)) is the pixel coordinates of the firstclass of interest).

p (a second camera parameter) of Calculation Formula 1 may berepresented with Calculation Formula 2. In addition, h_(side) (x, y) andh_(front) (x, y) may be represented with Calculation Formula 3 andCalculation Formula 4, respectively.

p=[Φθψt _(x) t _(y) t _(z)]^(T)   <Calculation Formula 2>

where, φ is a first parameter of the second camera, Φ is a secondparameter of the second camera, ψ is a third parameter of the secondcamera, t_(x) is a fourth parameter of the second camera, t_(y) is afifth parameter of the second camera, t_(z) is a sixth parameter of thesecond camera, and T is a transpose.

$\begin{matrix}{{h_{side}\left( {x,y} \right)} = {\begin{bmatrix}\frac{h_{00x} + h_{01y} + h_{02}}{h_{20x} + h_{21y} + h_{22}} \\\frac{h_{10x} + h_{11y} + h_{12}}{h_{20x} + h_{20y} + h_{22}}\end{bmatrix} = \begin{bmatrix}u_{side} \\v_{side}\end{bmatrix}}} & {< {{Calculation}{Formula}3} >}\end{matrix}$

where, h₀₀ to h₂₂ are calculated using a 3*3 homography matrixcalculated with the parameters of the second camera.

$\begin{matrix}{{h_{front}\left( {x,y} \right)} = {\begin{bmatrix}\frac{h_{00x} + h_{01y} + h_{02}}{h_{20x} + h_{21y} + h_{22}} \\\frac{h_{10x} + h_{11y} + h_{12}}{h_{20x} + h_{20y} + h_{22}}\end{bmatrix} = \begin{bmatrix}u_{front} \\v_{front}\end{bmatrix}}} & {< {{Calculation}{Formula}4} >}\end{matrix}$

where, h₀₀ to h₂₂ are calculated using a 3*3 homography matrixcalculated with parameters of the first camera.

The processor 140 may project the pixel coordinates of the first classof interest onto the second image and convert the pixel coordinatesusing Calculation Formula 1 to calculate the converted pixel coordinatesof the first class of interest.

The processor 140 may correct the parameters of the second camera suchthat a difference between the converted pixel coordinates of the firstclass of interest and the pixel coordinates of the second class ofinterest is minimized. According to an embodiment, the processor 140 mayiterate the calculation process in Calculation Formula 5 such that adifference “E” between the pixel coordinates of the first class ofinterest and the pixel coordinates of the second class of interest isminimized.

$\begin{matrix}{E = {\sum\limits_{u_{front},{v_{front} \in {ROI}}}\left\lbrack {{C_{front}\left( {u_{front},v_{front}} \right)} - {C_{side}\left( {W\left( {u_{front},v_{front},p} \right)} \right)}} \right\rbrack^{2}}} & {< {{Calculation}{Formula}5} >}\end{matrix}$

where, C_(front) is the first image, C_(side) is the second image,(u_(front), v_(front)) is the pixel coordinates of the first class ofinterest, and W is the warping function.

A more detailed description is made with reference to FIGS. 6 and 7 .

FIG. 6 is a calibration graph of a second camera according to anembodiment of the present disclosure, and FIG. 7 is a view showing achange in matching of a first image and a second image based on anincrease in the number of iterations according to an embodiment of thepresent disclosure.

As shown in FIG. 6 , the processor 140 may correct the parameters of thesecond camera by increasing the number of iterations of the calculationprocess in Calculation Formula 5 to minimize the difference (an error)between the pixel coordinates of the first class of interest and thepixel coordinates of the second class of interest.

FIG. 7 includes a first image 70 to a seventh image 82 showing thechange in the matching of the first image and the second image based onthe increase in the number of iterations according to an embodiment ofthe present disclosure. In an embodiment of the present disclosure, whenan initial value of the pixel coordinates of the first class of interestconverted using the warping function is projected onto the second image,as in the first image 70, a difference occurs between the second classof interest 30 of the second image and a converted first class ofinterest 24.

The processor 140 may iterate the calculation in Calculation Formula 520 times and continuously correct the parameters of the second camera,thereby minimizing the difference between the first class of interest 24converted as in the second image 72 to the seventh image 82 and thesecond class of interest 30 of the second image.

FIG. 8 is a view showing a camera calibration method of a vehicleaccording to an embodiment of the present disclosure.

As shown in FIG. 8 , the processor 140 may acquire the first image fromthe first camera 110 (S110 ). In addition, the processor 140 may acquirethe second image from the second camera 120 (S120).

The processor 140 may extract the first class of interest 20 from thefirst image acquired from the first camera 110 (S130). According to anembodiment, in S130, the processor 140 may set the region including theobject having the height of zero “0” in the predetermined region of thefirst image as the region of interest, and the processor 140 may extractthe pixels of the region of interest as the first class of interest. Forexample, the processor 140 may extract the line or the pattern paintedon the road surface as the first class of interest. The processor 140may obtain the pixel coordinates of the first class of interest.

In addition, the processor 140 may extract the second class of interest30 from the second image acquired from the second camera 120 (S140).According to an embodiment, in S140, the processor 140 may set theregion including the object having the height of zero “0” in thepredetermined region of the second image as the region of interest, andthe processor 140 may extract the pixels in the region of interest asthe second class of interest. For example, the processor 140 may extractthe line or the pattern painted on the road surface as the second classof interest. The processor 140 may obtain the pixel coordinates of thesecond class of interest.

The processor 140 may convert the pixel coordinates of the first classof interest extracted from the first image by projecting the pixelcoordinates onto the second image (S150). According to an embodiment, inS150, the processor 140 may convert the pixel coordinates of the firstclass of interest by projecting the pixel coordinates onto the secondimage using the warping function “W” (see, Calculation Formula 1), andcalculate the converted pixel coordinates of the first class ofinterest.

The processor 140 may correct the parameters of the second camera suchthat the difference between the converted pixel coordinates of the firstclass of interest and the pixel coordinates of the second class ofinterest is minimized (S160). According to an embodiment, the processor140 may iterate the calculation process in Calculation Formula 5 suchthat the difference between the pixel coordinates of the first class ofinterest and the pixel coordinates of the second class of interest isminimized.

FIG. 9 is a view showing a configuration of a computing system executinga method according to an embodiment of the present disclosure.

With reference to FIG. 9 , a computing system 1000 may include at leastone processor 1100, a memory 1300, a user interface input device 1400, auser interface output device 1500, storage 1600, and a network interface1700 connected via a bus 1200.

The processor 1100 may be a central processing unit (CPU) or asemiconductor device that performs processing on commands stored in thememory 1300 and/or the storage 1600. The memory 1300 and the storage1600 may include various types of volatile or non-volatile storagemedia. For example, the memory 1300 may include a ROM (Read Only Memory)1310 and a RAM (Random Access Memory) 1320.

Thus, the operations of the method or the algorithm described inconnection with the embodiments disclosed herein may be embodieddirectly in hardware or a software module executed by the processor1100, or in a combination thereof. The software module may reside on astorage medium (that is, the memory 1300 and/or the storage 1600) suchas a RAM, a flash memory, a ROM, an EPROM, an EEPROM, a register, a harddisk, a removable disk, and a CD-ROM. The exemplary storage medium iscoupled to the processor 1100, which may read information from, andwrite information to, the storage medium. In another method, the storagemedium may be integral with the processor 1100. The processor and thestorage medium may reside within an application specific integratedcircuit (ASIC). The ASIC may reside within the user terminal. In anothermethod, the processor and the storage medium may reside as individualcomponents in the user terminal.

The description above is merely illustrative of the technical idea ofthe present disclosure, and various modifications and changes may bemade by those having ordinary skill in the art without departing fromthe essential characteristics of the present disclosure.

Therefore, the embodiments disclosed in the present disclosure are notintended to limit the technical idea of the present disclosure but toillustrate the present disclosure. The scope of the technical idea ofthe present disclosure is not limited by the embodiments. The scope ofthe present disclosure should be construed as being covered by the scopeof the appended claims, and all technical ideas falling within the scopeof the claims should be construed as being included in the scope of thepresent disclosure.

The device and the method for calibrating the camera of the vehicleaccording to an embodiment of the present disclosure may correct theplurality of images acquired by the plurality of cameras to be convertedinto the same real-world coordinates, and provide the informationnecessary for the travel of the autonomous vehicle, thereby improvingtravel safety.

Hereinabove, although the present disclosure has been described withreference to exemplary embodiments and the accompanying drawings, thepresent disclosure is not limited thereto, but may be variously modifiedand altered by those having ordinary skill in the art to which thepresent disclosure pertains without departing from the spirit and scopeof the present disclosure.

What is claimed is:
 1. A device for calibrating a camera of a vehicle,the device comprising: a first camera configured to acquire a firstimage; a second camera configured to acquire a second image; and aprocessor configured to: extract a first class of interest from thefirst image; extract a second class of interest from the second image;project pixel coordinates of the first class of interest onto the secondimage to convert the pixel coordinates; and correct parameters of thesecond camera such that a difference between the converted pixelcoordinates of the first class of interest and pixel coordinates of thesecond class of interest is minimized.
 2. The device of claim 1, whereinthe processor is configured to: set a region including an object havinga height of zero “0” in a predetermined region of the first image as aregion of interest; and extract pixels of the region of interest as thefirst class of interest.
 3. The device of claim 1, wherein the processoris configured to: set a region including an object having a height ofzero “0” in a predetermined region of the second image as a region ofinterest; and extract pixels of the region of interest as the secondclass of interest.
 4. The device of claim 1, wherein the processor isconfigured to extract a line or a pattern painted on a road surface asthe first class of interest.
 5. The device of claim 1, wherein theprocessor is configured to extract a line or a pattern painted on a roadsurface as the second class of interest.
 6. The device of claim 1,wherein the processor is configured to convert the pixel coordinates ofthe first class of interest by projecting the pixel coordinates onto thesecond image using a warping function.
 7. The device of claim 1, whereinthe first camera and the second camera are disposed at differentpositions.
 8. A method for calibrating a camera of a vehicle, the methodcomprising: acquiring a first image from a first camera and acquiring asecond image from a second camera; extracting a first class of interestfrom the first image and extracting a second class of interest from thesecond image; projecting pixel coordinates of the first class ofinterest onto the second image to convert the pixel coordinates; andcorrecting parameters of the second camera such that a differencebetween the converted pixel coordinates of the first class of interestand pixel coordinates of the second class of interest is minimized. 9.The method of claim 8, further comprising: setting a region including anobject having a height of zero “0” in a predetermined region of thefirst image as a region of interest; and extracting pixels of the regionof interest as the first class of interest.
 10. The method of claim 8,further comprising: setting a region including an object having a heightof zero “0” in a predetermined region of the second image as a region ofinterest; and extracting pixels of the region of interest as the secondclass of interest.
 11. The method of claim 8, further comprising:extracting a line or a pattern painted on a road surface as the firstclass of interest.
 12. The method of claim 8, further comprising:extracting a line or a pattern painted on a road surface as the secondclass of interest.
 13. The method of claim 8, further comprising:converting the pixel coordinates of the first class of interest byprojecting the pixel coordinates onto the second image using a warpingfunction.
 14. The method of claim 8, wherein the first camera and thesecond camera are disposed at different positions.